Magnetic resonance imaging (MRI) is well established as a technique for elucidating structure, and is used, for example, to discriminate between normal and pathological tissue. The MRI image contrast depends on differences in parameters such as the proton density, the longitudinal relaxation time T1, and the transverse relaxation time T2 for different media. MRI has further been extended to imaging in which parameters such as the magnitude and phase of the transverse magnetization, the magnitude of the longitudinal magnetization, and the resonance frequency (related to spectroscopy) are related to functions such as molecular displacements (e.g., flow and diffusion). Further, MR has been extended to measure tensor-valued diffusion rates.
Diffusion along a given axis is typically measured by placing a pair of diffusion sensitizing gradient pulses in the same axis in the magnetic resonance (MR) pulse sequence. The gradient pulses impose position-dependent phases on water protons that are equal in magnitude but opposite in sign, and therefore cancel for stationary spins. However, for protons that move between the two gradient pulses, a finite net phase is accumulated. The sum of all phases from all protons results in attenuation of the MR signal due to interference effects. The magnitude of signal attenuation is dependent on the diffusivity of water, and the width, separation and amplitude-of the gradient pulses. In a generalized case, where the diffusivity may differ in different directions, a diffusion tensor matrix notation is used.
The above-noted MRI techniques generate large multivalued datasets representing a three dimensional (3D) structure, such as human tissue. In order to be of use, the datasets need to be converted to a form that can be displayed to a user, such as a physician or other health care professional. The process of converting the datasets to a displayable form can be referred to as volume rendering, as compared to surface rendering techniques that operate, typically, with tessellated surface representations of three dimensional objects.
In addition to datasets derived from MRI, multivalued datasets can be generated by modeling and simulations, and with other imaging modalities such as PET, OPT, CT, EEG and MEG.
Representative U.S. Patents that pertain at least in part to the generation of images derived from MRI-generated datasets include the following: U.S. Pat. No. 5,539,310, “Method and System for Measuring the Diffusion Tensor and for Diffusion Tensor Imaging”, by Basser et al.; U.S. Pat. No. 5,969,524, “Method to Significantly Reduce Bias and Variance of Diffusion Anisotrophy Measurements”, by Pierpaoli et al.; U.S. Pat. No. 6,441,821, “Method and Apparatus for Displaying Three-Dimensional Image by Use of Tensor Rendering”, by Nagasawa; U.S. Pat. No. 6,463,315, “Analysis of Cerebral White Matter for Prognosis and Diagnosis of Neurological Disorders”, by Klingberg et al.; U.S. Pat. No. 6,526,305, “Method of Fiber Reconstruction Employing Data Acquired by Magnetic Resonance Imaging”, by Mori; and U.S. Pat. No. 6,529,763, “Imaging of Neuronal Material”, by Cohen et al.
By example, U.S. Pat. No.: 6,526,305 (Mori) describes a method of creating an image of brain fibers that includes exposing the brain fibers to a MRI process. The data acquired from the MRI includes the acquisition of diffusion-weighted images that are later employed to calculate an apparent diffusion constant at each pixel along more than six axes. The data is introduced into a microprocessor that calculates six variables in a diffusion tensor and obtains a plurality of eigenvalues and eigenvectors. This is accomplished by employing a diffusion sensor that is diagonalized to obtain three eigenvalues and three eigenvectors, with the six values being subjected to further processing to generate imaging information representing the properties of the fibers. The process includes the initiation of fiber tracking by selecting a pixel for initiation of tracking, connecting pixels and effecting a judgement regarding termination of the pixel tracking in each direction based upon the randomness of the fiber orientation of the adjacent pixels.
Other publications related to the problem of imaging multivalued data sets, in particular diffusion tensor field visualization, vector field visualization, hardware-accelerated volume rendering, and thread rendering are now discussed.
Diffusion Tensor Field Visualization
There are several approaches to visualizing diffusion tensor imaging (DTI) datasets. Since a diffusion tensor is a symmetric matrix with positive eigenvalues, an ellipsoid is a natural geometric representation for it. Pierpaoli, C., and Basser, P. 1996, “Toward a Quantitative Assessment of Diffusion Anisotropy”, Magnetic Resonance Magazine, 893-906, used a two dimensional (2D) array of ellipsoids to visualize a 2D diffusion tensor field. To give a more continuous visual appearance, Laidlaw, et al. normalized the ellipsoids, “Visualizing Diffusion Tensor Images of the Mouse Spinal Cord”, In Proceedings IEEE Visualization '98, IEEE, 127-134. Additionally, Laidlaw et al. employed concepts from oil painting, i.e., mapping data components onto brush strokes and building up the strokes in multiple layers, to represent more of the data, creating a second type of 2D visualization showing all of the multiple values simultaneously.
However, these 2D-based techniques do not generalize well to three dimensional (3D) representations. Placing the ellipsoids in 3D to visualize a 3D dataset has at least two drawbacks. First, placing an ellipsoid at every data point in three dimensional space obscures all layers of ellipsoids except the outermost layer. Second, continuity cannot be shown in an ellipsoid representation, and without continuity information, neural connectivity is difficult to visualize and understand.
Kindlmann et al., “Hue-Balls and Lit-Tensors for Direct Volume Rendering of Diffusion Tensor Fields”, In Proceedings IEEE Visualization '99, IEEE, 183-189, overcame the problem of obscuring data points in 3D with a direct volume rendering approach: to every data point they assign a certain opacity and color based on the underlying diffusion tensor dataset, using the concept of a “barycentric map” for the opacity and “hue-balls” and “lit-tensors” for coloring and lighting. However, the use of this approach does not solve the problem of accurately and unambiguously locating anatomically distinct regions of the brain, and understanding the underlying connectivity.
Basser et al., “In Vivo Fiber Tractography Using DT-MRI Data”, Magnetic Resonance in Medicine 44, 625-632, calculated the trajectories of neural fibers in brain white matter that were generated from the diffusion tensor field by integrating along the eigenvector with the largest eigenvalue. Zhang et al., “Streamtubes and Streamsurfaces for Visualizing Diffusion Tensor MRI Volume Images”, in IEEE Visualization 2000, used this method to generate streamtubes to visualize continuous directional information in the brain (the streamtubes are calculated during a preprocessing step). The use of streamtubes aids in solving the problem of representing continuity, but at the cost of the loss of structural information in those areas without stream tubes.
Of the extensive amount of effort that has been directed to creating effective vector field visualizations, the following two publications are believed to be of the most interest to this invention. Interrante, V., and Grosch, C., “Strategies for Effectively Visualizing 3D Flow with Volume LIC”, In Proceedings of the conference on Visualization '97, 421, visualized 3D flow with volume line integral convolution (LIC). Adapting artistic techniques, they introduced 3D visibility-impeding halos to improve the perception of depth discontinuities in a static volume renderer. As they demonstrated with off-line rendering, halos improve depth perception and help make complex 3D structures easier to analyze for the viewer. In addition, Zöckler et al., “Interactive visualization of 3D-Vector fields using illuminated streamlines”, in IEEE Visualization '96, IEEE, ISBN 0-89791-864-9, introduced illuminated field lines to visualize 3D vector fields.
Hardware-Accelerated Volume Rendering
3D volumes are often stored in a texture memory in hardware-accelerated volume rendering systems. Data are then mapped onto slices through the volume and blended together to render an image. Cabral et al., “Accelerated Volume Rendering and Tomographic Reconstruction Using Texture Mapping Hardware”, in ACM Symposium On Volume Visualization, ACM, introduced a 2D texture approach for such rendering, where three stacks of 2D slices are generated through the volume, one perpendicular to each coordinate axis. As the viewpoint changes, the renderer chooses the “best” stack to render. This approach exploits hardware-based bilinear interpolation in the plane of the slice. A drawback of this approach, however, is that three copies of the volume must be stored in the texture memory.
Van Gelder, A. and Kim, K., “Direct volume rendering with shading via three-dimensional textures”, in 1996 Volume Visualization Symposium, IEEE, 23-30, ISBN 0-89791-741-3, avoided the redundant data copies by rendering with 3D textures. Generally view-aligned slices are used, exploiting a trilinear interpolation capability in the hardware.
Several volume-rendering implementations employ commercial graphics cards or distributed hardware. However, these approaches do not target the visualization of multivalued datasets.
Kniss et al., “Interactive Volume Rendering Using Multi-Dimensional Transfer Functions and Direct Manipulation Widgets”, In IEEE Visualization 2001, IEEE, used interactive transfer functions operating on directional derivative values to select boundaries in scalar-valued datasets. Engel et al., “High-Quality Pre-Integrated Volume Rendering Using Hardware-Accelerated Pixel Shading”, in Siggraph/Eurographics Workshop on Graphics Hardware 2001, ACM, described an approach that rendered more quickly, with fewer slices, by using additional precalculated volumes.
Lum, E. B., and Ma, K. L., “Hardware-Accelerated Parallel Non-Photorealistic Volume Rendering”, In Proceedings of the Non-Photorealistic Animation and Rendering conference 2002, implemented a hardware-accelerated parallel non-photorealistic volume renderer that uses multi-pass rendering on consumer-level graphics cards. Their system emphasizes edges or depth ordering using artistically motivated techniques.
Hair, Fur, and Thread (Filament) Rendering
Kajiya, J. T., and Kay, T. L., “Rendering fur with Three Dimensional Textures”, in Proceedings SIGGRAPH '89, ACM, pp. 271-280, introduced the concept of texels to render realistic-looking fur. Texels are 3D texture maps in which both a surface frame (normal and tangent) and the parameters of a lighting model are distributed throughout a volume, thereby representing a complex collection of surfaces with a volumetric abstraction. Kajiya and Kay also developed a Phong-like lighting model for fur.
Lengyel, J. E., “Real-Time Fur”, in Eurographics Rendering Workshop 2000, ACM, 243-256, used a volumetric texture approach to render short threads in real time that is based on a stack of partially transparent 2D texture layers. In a preprocessing step, procedurally defined threads are filtered into layers and blended together. In Lengyel's approach, lighting calculations are performed at run time using Banks' hair-lighting model.
As can be appreciated, the creation of comprehensive and accurate visualizations for exploring 3D multivalued data presents a number of challenges. A first challenge is to create visualizations in which the data nearer to the viewer does not excessively obscure that farther away. A second challenge is to represent many values and their interrelationships at each spatial location. As an example, neuroimaging data and flow data may each have many primary values at each point. The multiple values exacerbate the obscuration problem, not only because more values must be shown at each spatial location, but also because important relationships among the different data values require the use of more of finite visual bandwidth. A third challenge is to convey relationships among different spatial locations. A further challenge is to render the large datasets interactively.
A need thus exists to address and overcome these and other challenges related to the volumetric imaging of multivalued, large datasets. Prior to this invention, however,this need was not adequately met.